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AUTOMATED VERIFICATION AND INSPECTION DEVICE 
FOR SEQUENTIALLY INSPECTING MICROSCOPIC CRYSTALS 
The present invention relates to automated inspection and verification devices, and in 
particular to devices for sequentially inspecting microscopic crystals. 

BACKGROUND OF THE INVENTION 
The determination of the three dimensional atomic structure of matter is one of the 
most important areas of pure and applied research. One way in which the three 
dimensional atoroic structure of matter can be determined is through X-ray 
crystallography. X-ray crystallography utilizes the diffraction of X-rays from crystals 
in order to determine the precise arrangement of atoms within the crystal. The result 
may reveal the atomic structure of substances such as metal alloys, deoxyribonucleic 
acid (DNA), or the structure of proteins. 

There are very important benefits to knowing the accurate molecular structure of a 
protein crystal. For example, once the molecular structure is known, a drug designer 
can more effectively develop effective therapeutic agents and drugs. However, 
despite its promises, X-ray crystallography is limited by the fact that it is very 
difficult to grow successful crystals. 

Prior Art Method of Growing Crystals 
Protein crystals are commonly grown in the wells of micro-well plates. A micro-well 
plate is also known as a micro-titer plate or a microplate. Micro-well plates typically 
come with either 24, 48, 96, 384 or 1536 wells. A 96-well micrp-well plate is shown 
in detail m FIG. 2. There are a variety of methods in which protein crystals may be 
grown. Five common ways are simimarized below. 

Hanging Drop Method 
One of the main techniques available for growing crystals, known as the hanging-drop 
or vapor diffusion method, is a method wherein a drop of a solution containing 
protein is appUed to a glass cover slip and placed upside down in an apparatus such as 
a vapor diffusion chamber where conditions lead to supersaturation in the protein drop 
and the initiation of precipitation of the protein crystal. 
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Sitting Drop Method 

Another method is the sitting drop method where the drop sits in a small well adjacent 
the growing solution instead of hanging over it This method provides a more stable 
drop and location. 

Aqueous Drop in Oil Method 
Another method is the aqueous drop in oil method. The drop is placed in a micro- 
well and is covered with an oil based solution. The drop stays at the bottom of the 
well as the crystal grows. 

Dialysis Method 

In another method referred to as the dialysis method (also called microbatch 
crystallization), the protein solution is contained within a semi-permeable size 
exclusion membrane and then placed in a solution of fixed pH and precipitant 
concentration. As the precipitant dififijses through the membrane into the protein 
conq)artment, the solubility of the protein is reduced and crystals may form. 

Gel Crystal Growth Method 
This method involves the placement of a gel into the end of small diameter glass 
capillaries. After the solutions have gelled, a protein solution is placed into one end 
(top) of the capillary and the other end is submerged in a solution of precipitating 
agent. If the conditions are ^propriately selected, crystal growth occurs at a point in 
the gel where the protein and precipitating agent reach the proper concentrations as 
the solutions slowly mix by diffusion. Since this is a difiusion lintiited process, it thus 
only occurs after an extended period of time. Crystals however, grown by this method 
are often larger and of higher quality. 

Regardless of the method chosen, protein crystal growth is a very delicate and time- 
consuming process. It can take several days to several months before crystals of 
sufficient size and quality are grown and ready for x-ray crystallogr^hy. The current 
minimum size that is typically stated is a crystal of at least 50 microns thick by 100 
microns in extent. The protein crystal growing environmental conditions need to be 
rigorously maintained, ftom the chemistry, to the surrounding air hxmiidity and 
temperature, cleanliness to prevent contamination, and even lighting conditions. A 
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protein crystallographer working with unknown protein families may only be about 
5% successful in growing proper sized quality crystals. With this success rate, for 
example, a 96-well micro-well plate may only have 5 wells in which good crystals are 
growing. 

Prior Art Inspection of Crystal Growth 
Currently, a laboratory technician, or operator, aided by a microscope and a 
laboratory notebook manually inspects crystals grown in micro-well plates. To 
inspect a micro-well plate, a laboratory technician dons a clean-room gown suit and 
enters a cold room in which the crystals are growing. The technician then puts a 
micro-well plate underneath the microscope and examines each well in the micro-well 
plate until all of the wells in the micro-well plate have been inspected. The technician 
then makes a mental judgement as to how he shall classify (also known as "score") 
the crystal. For example, the technician may feel that he is observing an image that 
shows "grainy precipitation" or *\igly precipitation". Or, he may feel that the unage 
shows "no crystal growth". The technician then records the classification into a 
laboratory notebook. 

The above system is riddled with opportunities for human error. An operator, 
manually inspecting a 96-well micro-well plate will take approximately 5 to 20 
minutes depending on the skill of the operator and the number of wells that contain 
mteresting features, microcrystals, or crystals. The operator may be subject to 
physical fatigue, suffer eyestrain, and may be uncomfortably cold m the temperature 
controlled and generally higji humidity room. The operator can be tired and confused 
and can easily make errors in manually recording data in the notebook. For example, 
the operator may observe crystal growth at well H5 (FIG. 2), but incorrectly record in 
the notebook that the crystal growth was at well H6. Additional transcription errors 
may occur when the data is transferred to a computer database. 

Research efforts are underway to try to solve the above problem, but they are 
inadequate for the needs of the industry. One such effort is described in Jurisica et al. 
*ThteUigent Decision Support for Protein Crystal Growth" IBM systems Journal, Vol. 
40, No 2, 2001 . Another such effort is described at the Website www.dsitecLcom. 
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What is needed is a better device and method for inspecting crystal growth. 

SUMMARY OF THE INVENTION 
The present invention provides a naethod and device for inspecting and classifying a 
plurality of microscopic crystals. An indexing device sequentially places microscopic 
crystals in camera-view of a camera. The camera takes images of the microscopic 
crystals and then transfers the images to a computer where the are received* The 
computer then classifies the images. The computer is also programmed to control the 
indexing device and the camera Li a preferred embodiment, the microscopic crystals 
are protein crystals that have been grown in the wells of micro-well plates. Also, 
preferably, the computer is programmed to automatically classify the images. 

BRIEF DESCRIPTION OF THE DRAWINGS 
FIG. 1 shows a preferred embodiment of the present invention. 
FIG. 2 shows a micro-well plate. 
FIG. 3 shows a top view of the fixture plate. 

FIGS. 4 and 5 show top views of micro-well plates on the fixture plate. 

FIG. 6 shows a block diagram of a preferred embodiment of the present invention. 

FIG. 7 shows a preferred monitor. 

FIGS. 8-10 and 18-25 show steps in the sequence of operations of a preferred 

embodiment of the present invention, 

FIG. 1 1 shows hanging drops of liqxiid in a micro-well plate. 

FIG. 12 shows an example of aqueous drop in oil protein crystallization. 

FIG. 13 shows a top view of a micro-well plate on the fixture plate. 

FIG. 14 shows a side view of the ligjit source shining upwards onto a micro-well 

plate. 

FIG. 15 shows a magnified view of two wells of a micro-well plate, wherein each 
well has a drop of liquid. 

FIGS, 16 and 17 show a detail view of the drops of liquid shown in FIG. 15. 

FIG. 26 shows a preferred monitor screen after a run has been completed 

FIGS. 27 and 28 show details of other preferred monitor screens. 

FIG. 29 shows a hanging drop of liquid with crystal growth. 

FIG. 30 shows a prefOTred embodiment of the present invention. 

FIG. 3 1 shows a flowchart of an auto-focus subroutine of the present invention. 
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FIG 32. shows a flowchart of a focus value subroutine. 
FIG. 33 shows a flowchart of the auto score aad classify subroutine. 
FIGS. 34a - 34d show flowcharts of the classify subroutine. 
FIG. 35a -35b show the sub-classification of the crystal class. 
FIG. 36 shows the main program flow. 

FIG. 37 illustrates a side view illustrating dual filters in the light path. 

FIG. 38 illustrates a top view of the drive mechanism for the rotatable linear polarized 

filter. 

FIG. 39 illustrates a top view of a second filter wheel. 

FIG. 40 shows the connectivity of another preferred embodiment. 

DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT 
A detailed description of a preferred embodiment of the present invention can be 
described by reference to the drawings. 

FIG. 1 shows a preferred embodiment of the present invention. Micro-well plates 
125A - 125F are placed on fixture plate 129. La a preferred embodiment, each micro- 
well plate has 96 wells. Each well has a drop of Uquid in which microscopic protein 
crystals may be growing. Computer 105 automatically controls linear actuators 115, 
150 and 160. Linear actuator 115 moves fixture plate 129 along the x-axis. Linear 
actuator 150 moves moving base 154 along the y-axis, and linear actuator 160 moves 
moving plate 162 along the z-axis. n The computer coordinates the movement of the 
linear actuators to properly position cameras 155 and 135 above each well of each 
micro-well plate 125A - 125F in sequence. Preferably, cameras 155 and 135 are 
high-resolution 2/3 inch CCD cameras, with 1,300 horizontal pixel elements by 1,030 
vertical elements. Each element is preferably 6.7 microns square with a sensing area 
of 8.7 mm by 6.9 mm. Cameras 155 and 135 take images of each well and transmit 
the images to computer 105 where they are digitized into a image data array of 
approximately 1,296 horizontal pixels by 1,000 vertical pixels by 8 bits of gray level 
representing the intensity (0 to 255) of each pixel. These digitized images are 
automatically recorded in the database of computer 105 and can be analyzed and 
scored by an operator via monitor 620, The digitized images may be fiirther 
processed, analyzed, and the contents of the individxxal well scored by conq)uter 105 
executing program instructions to perform calculations on the image data array. In a 
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preferred embodiment, computer 105 is connected via a commimication/control line 
to a computer network. In this manner, the present invention can be controlled firom a 
remote computer. Likewise, images and data can be transmitted to the remote 
computer. 

Sequence of Operation of a Preferred Embodiment 
Micro-well plates loaded onto the System 
As shown in FIG. 1, six 96-well micro-weU plates 125A - 125F have been placed 
(either by an operator or by an external loading robot) onto fixture plate 129. A 96 
well micro-well plate 125A is shown in FIG. 2. Micro-well plate 125A has wells 
labeled Al through H12 and a bar code label 220. Micro-well plate 125A is available 
firom Nalge-Nunc International, with U.S. offices in Rochester, NY, Fixture plate 129 
will hold 24, 48, 96, 384, or 1536 well micro-well plates since the micro-well plate 
extemal width and length are fairly standard in the industry. The 96 well micro-well 
plate will be used to illustrate the present invention. 

FIG. 3 shows a top view of fixture plate 129 just prior to loading micro-well plates 
125A - 125F. Fixture plate 129 has six cutout sections 131 that are just slightly 
smaller in length and width than micro-well plates 125A - 125F. 

FIG. 4 shows a top view of micro-well plates 125A - 125F unmediately after they 
have been placed on fixture plate 129. 

After the operator has placed micro-well plates 125A - 125F onto fixture plate 129 as 
shown in FIG. 4, he enters the command into computer 105 (FIG. 1 and FIG. 6) to 
expand detents 510 and 520 (FIG. 5). The expansion of detents 510 and 520 firmly 
secures micro-well plates 125A - 125F against plate stops 530. 

Recording ttie Bar Code Information for the Micro-well plates 
Computer 105 (FIG. 1) has been programmed to accept inputs firom an operator. FIG. 
7 shows a display representing micro-well plates 125 A - 125F on the screen of 
monitor 620. In FIG, 7, the operator has mouse clicked on bars 622A - 622F, which 
has caused them to turn green. By chcking on bars 622A - 622F, the operator has 
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selected corresponding micro-weU plates 125A - 125F to W. The operator sends 
the command to run the selected micro-well plates by cHcldng on run bar 623. 

In no. 8, the operator has given the command to run the selected micro-well plates. 
Micro-well plate 125A has been moved to a position underneath cameras 155 and 
135. By the utilization of plate sensor transmitter/receiver 186 and reflector 188, 
information is sent to computer 105 reporting that micro-well plate 125A is in 
position underneath the cameras. Plate sensor transmitter/receiver 186 is fixed to 
support 189 and is aUgned to sense whenever a micro-well plate breaks a beam of 
light emitted by transmitter/receiver 186 and reflected by a reflector 188. Reflector 
188 is mounted on support 191 on the opposite side of the linear actuator 115. The 
plate sensor transmitter/receiver 186 and reflector 188 are preferably model # E3T- 
SRl 3 available ftom Western Switch Controls of Santa Ana, CA. 

Bar code reader 175 is also mounted to support 189 and is positioned to view bar- 
code identity IsOjel 220 (FIG. 2) attached to micro-well plate 125A when it is 
positioned underneath cameras 155 and 135. Bar-code reader 175 is preferably model 
# BL601 available from Keyence Corporation of America of Newark, NJ. Bar-code 
reader 175 communicates with computer 105 via a communication line. Information 
encoded into label 220 preferably includes: the plate serial number, the plate type 
(i.e., 24-welI, 48-well, 96-well, 384-weU, or 1536-well micro-weU plate), and the well 
type (i.e., square, or rounded, hanging drop, sitting drop, constrained sitting drop). 

The information from plate sensor transmitter/receiver 186 and bar-code reader 175 is 
transmitted to computer 105 and stored for later use during the camera inspection and 
information acquisition phase. 

In FIG. 9, linear actuator 115 has moved fixture plate 129 so that micro-weU plate 
125B is underneath cameras 155 and 135. In a fashion sinular to that described above 
with regards to micro-well plate 125A, information is transmitted from plate sensor 
transmitter/receiver 186 and bar-code reader 175 to computer 105 and stored for later 
use during the camera inspection and information acquisition phase. 
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The above described sequence continues until aU micro-weU plates 125A - 125F have 
been sensed and recorded by plate sensor transmitter/receiver 1 86 and bar-code reader 
175. 

Then, as shown in FIG. 10, linear actuator 115 moves micro-well plate 125A so ttiat it 
is undemealh lens 165 of camera 155. Motor 130 of linear actuator 160 moves 
moving plate 162 upward and/or downward as necessary to properly focus lens 165 
on the drop of hanging Uquid over weU Al. Preferably, lens 165 is set at a 
predetermined zoom. 

fcspection of Crystals 
Determining the Position of tiie Drop of Liquid within Each WeU 
An operation to inspect each well to detennine the position of each hanging drop of 
Uquid is performed on micro-weU plate 125A after it has been moved to the position 
shown in FIG. 10. 

FIG. 11 shows a cross section side view of weUs Al - El of micro-weU plate 125A. 
hi a preferred embodiment, an attempt has been made to grow protein crystals in the 
hanging drops in each oftheweUs of micre-weU plate 125A. FIG. 11 shows hanging 

drops al, pi, %i, 51, and si. 

The preferred method for protein crystal growth is the hangmg drop method. The 
hangmg drop method (also known as vapor diffiision) is probably the most common 
method of protein crystal growtii. As explained in the background section, a drop of 
protein solution is suspended over a reservok containing buffer and precipitant. Water 
diffiises fiom the drop to the solution leaving the drop with optimal crystal growth 
conditions. 

hi FIG. 10, lens 165 of camera 155 is over weU Al of micro-weU plate 125A (FIG. 2, 
FIG. 1 1, and ¥IG. 13). FIG. 13 shows a top view of micro-weU plate 125A positioned 
on fixture plate 129. 
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FIG. 14 shows a side view of micro-weU plate 125A positioned on fixture plate 129. 
Support 191 with embedded Ught source 194 is positioned to the side of fixture plate 
129. light from light guide 195 is directed upward through cutout 131 (also shown in 
FIG. 3). Light guide 195 is positioned between fixture plate 129 and plate 127 such 
that both plates can move around the Ught guide 195 without interference. As 
explained above, fixture plate 129 has cutouts 131 (FIG. 3) that are smaller than the 
micro-weU plates 125 and located under each well plate, such that Ught from Ught 
guide 195 can be projected through the weU plates when they are brought into 
position for inspection. In the preferred embodiment, Ught source 194 is model # 
A08925 fiber-optic backUght available from Aegis Electionics of Carlsbad, CA. 

Camera 155 (FIG. 10) inspects well Al and transmits an image to computer 105 for 
digitization. As described above, camera 155 preferably (FIG. 10) inspects well Al at 
a IX magnification so that every 6.7 micron square pixel represents approximately 6.7 
square microns on tiie object being measured, allowing for some smaU geometric 
distortions caused by the lens 165. Computer 105 has been programmed to digitize 
the camera image and tiien by utiUzing vision software determines a position within 
weU Al for flie drop of Uquid hanging from grease seal 361. The position of tiie drop 
of Uquid is recorded for later use onto the hard drive of computer 105 and to a 
memory location within the computer. 

In a preferred embodiment, tiie vision software used to determine tiie position of flie 
drop of Uquid uses a software routine algoritimi called mvt_blob_find &om a 
collection of image processing tools caUed MVTools. MVTools is available from 
Coreco Imaging, US Office in Bedford, MA. 

After recording tiie position of tiie drop of Uquid hanging from grease seal 361 in weU 
Al, Unear actiiator 1 15 moves fixture plate sU^tiy to tiie left so tiiat lens 165 is over 
weU Bl (FIG. 13). In a fa^on similar to tiiat described for weU Al, tiie position of 
tiie drop of Uquid hanging from grease seal 361 m weU Bl is recorded on tiie hard 
drive of computer 105 and in computer memory. For example, as shown in FIG. 15, 
computer 105 will record fliat drop of Uquid al is towards tiie upper left-hand 
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quadrant of weU Al. Likewise, fke position of drop of Uquid pi is recorded onto the 
database of computer 105 as being in the lower right-hand quadrant of well Bl. 

In this manner, positions of the drops of Uquid are recorded for cells Al - H12. In 
HG. 18, linear actuator 115 has moved fixture plate 129 so that well HI is under lens 
165. 

In no. 19, linear actuator 115 has moved fixture plate 129 to the left and linear 
actuator 150 has moved moving base 154 sUghfly rearward so that lens 165 is over 
well A2 of micro-well plate 125A (FIG. 13). 

In a manner similar to that described above, positions of the drops of Uquid are 
recorded for cells A2 - H12 (FIG. 2, FIG. 13). In FIG. 20, Unear actuator 115 has 
moved fixture plate 129 to the left and Unear actuator 150 has moved moving base 
154 rearward so that well H12 is under lens 165. 

After positions of the drops of Uquid are recorded for cells Al - H12 for micro-weU 
plate 125A, Unear actuator 1 15 moves fixture plate 129 and Unear actuator 150 moves 
moving base 154 so that ceU Al of micro-weU plate 125B is underneath lens 165 
(FIG. 21). 

hi a manner sunilar to that described above, positions of the drops of Uquid are 
recorded for cells Al - H12 for each micro-well plate 125A - 125F. hi FIG. 22, Unear 
actuator 115 has moved fixture plate 129 and Unear actuator 150 has moved moving 
base 154 so that weU H12 of micro-weU plate 125F is under lens 165. 

Recording the Image of the Drop of Liquid within Each WeU 
An operation to inspect each hanging drop is performed at a higher magnification 
using camera 135 with its zoom lens 145 capable of magnifications of 2.5X to lOX 
corresponding approximately to digitized pixels representing 2.68 microns square ( at 
2.5X) to 0.67 microns square (at lOX). This inspection is done for the purpose of 
determining whether protein crystals have grown. Zoom motor 192 controls the 
degree of zoom for zoom lens 145. Using data representing the position of tiie drop 
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Of Uquid within each weU obtained during the inspect-weU sequence, computer 105 
(HG. 1) automaticaUy transmits a signal to linear actuators 115 and 150 to position 
lens 145 directly over the drop of Uquid within each weU. For example, in FIG. 23 
lens 145 is positioned over the top of weU Al of micro-well plate 125 A. Using the 
positioning data earUer obtained, lens 145 is precisely positioned so that it is able to 
zoom in on drop of Uquid al (FIG. 13). FIG. 16 shows a magnified view of drop of 
Uquid al. In FIG. 23, motor 130 of Unear actuator 160 has moved moving plate 162 
upward and/or downward as necessary to properly focus lens 165 on drop of Uquid 
al. Zoom motor 192 has manipulated lens 165 to obtain the desired degree of zoom. 
Camera 135 inspects well Al and transmits a signal representing the magnified image 
of the hanging drop of Uquid to computer 105. The images are stored on computer 
105 temporarily in memory for immediate analysis and on hard disk for later analysis. 

In a similar fashion, Unear actuators 115. 150 and 160 and zoom motor 192 operator 
to properly position and magnify zoom lens 145 over each hanging drop of Uquid to 
obtain desired focus and magnification for image data storage. For example, FIG. 17 
shows a magnified view of hanging drop of liquid pi. 

In a manner similar to that described above during the inspect-weU sequence, 
magnified images of the drops of Uquids (similar to those shown in FIGS. 16 and 17) 
are recorded for cells Al - H12 for micro-weU plate 125A (FIG. 2, FIG. 13). in FIG. 
24, linear actuator 115 has moved fixture plate 129 and Unear actuator 150 has moved 
moving base 154 so that weU H12 of micro^weU plate 125A is under lens 165. 

Hien, the sequence is repeated for micro-weU plates 125B - 125F so that magnified 
images of the hanging drops of liquids are recorded for each ceU Al - H12 for micro- 
weU plates 125B - 125F. In FIG. 25, the sequence has ended for micro-weU plates 
125A - 125F. Linear actuator 115 has moved fixture plate 129 and Unear actuator 
150 has moved moving base 154 so that weU H12 of micro-weU plate 125F is under 
lens 145. 
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Manual Scoring the Drop of Liquid within Each Well 
After micro-well plates 125A - 125F have been run, monitor 620 will appear as 
shown in no. 26. In HG. 26. six images representing micro-well plates 125A - 
125F appear on the screen. Above each image is a message •'Run Con^" indicating 
that image data for hanging drops of liquid has been transferred into computer 105. 
Beneath each image are buttons 710 - 715 marked "S". By mouse cUcldng on any 
button 710 - 715, the operator may manually score for successfid crystal formation 
each magnified image of each hanging drop of liquid. 

For example, in HG. 26, the operator can mouse cUck on button 710 to score micro- 
well plate 125A. 

In FIG. 27, the operator has mouse cUcked on the circle representing weU Al of 
micro-weU plate 125A. This has caused a magnified image to be displayed of drop of 
Uquid al in screen section 716. The operator has com:luded that there are no crystals 
m drop of Uquid al and has therefore mouse cUcked on button 717 for "NO 
CRYSTAL". On the display screen, this has caused the circle representing weU Al of 
micro-well plate 125A to turn red. 

In FIG. 28, the operator has mouse cUcked on the circle representing well Bl of 
micro-well plate 125A. This has caused a magnified image to be displayed of drop of 
Uquid pi in screen section 716. The operator has conceded that there are crystals in 
drop of Uquid pi and has therefore mouse cUoked on button 718 for "CRYSTAL". 
On the display screen, this has caused the circle representing weU Bl of micro-weU 
plate 125A to turn greoi. 

In a sunilar fashion, the above scoring procedure is repeated until all weUs Al - H12 
for micro-well plates 125 A - 125F have been scored as either red (NO CRYSTAL) or 
green (CRYSTAL). 

DataUtiUzation 

Once micro-weU plates USA -125F have all been scored, the operator has at his easy 
disposal a database that contains the identity of each micro-well plate that was 
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inspected along with a score summarizing whether crystal formation occurred for 
each weU in the mido-weU plate. The automated and efficient manner in which the 
operator is able to acquire his contrasts with the prior art method of laboriously 
inspecting each well with a microscope and the handwriting the results into a 
notebook. 

For example, to score six 96-well micro-weU plates utilizmg the present invention 
should take ^proximately no more than 10 to 15 minutes. 

In contrast, the prior art method of inspecting six 96-well micro-weU plates with a 
microscope and the handwriting the results into a notebook will take approximately 
30 to 1 00 minutes depending on the conditions discussed in the background section in 
addition to the time required to transcribe the results into a computer database. Plus, 
as previously explained in the background section, manual inspection and scoring is 
subject to a relatively high risk of human error. 

Second Preferred Embodiment 
In a second preferred embodiment, the depth of view of camera 135 is approximately 
50 to 100 micrometers. The crystal in the drop of Uquid may be larger than the depth 
of view or there may be crystals growing at various levels within the hanging drop of 
Uquid, as shown in FIG. 29. Iberefore. in the second preferred embodiment, lens 145 
is focused at multiple di^ent levels 721 - 724 and a set of images are recorded at the 
different levels so that the entire crystal may be analyzed. 

Specimen Auto-Focus 
The third preferred embodiment of the present invention utilizes a specimen auto- 
focus subroutine 300 (FIG. 31). Subroutine 300 ensures that the specimen within the 
micro-well is in-focus at the desired zoom (or magnification ratio) of image lens 145. 
Utilizing the auto-focus feature, the present invention causes camera 135 to take a 
number of images defined by a Number_of_Z_SUces. Typically, there are between 5 
and 10 sUces separated in the Z-axis from one another by a Z_Stq)_Size. The typical 
step size is 0.05mm to 0.25 mm. The sUces preferably start at a Z-Axis location 
defined by a Start_Z value, which is typicaUy at the bottom of the cover-slip on the 
micro-well plate. During specimen auto focus initialization 302, input data 304 is 
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received. The area of interest window within the images is further defined in the 
input data 304 by a X_Window_Center and a Y_Window_Centffl- plus an XJWidth 
and a Y_Width. Initial settings 306 for the routine 300 are the starting value of a 
counter N. a Best_Focus, a Best_Z, and a Focus_Error. The inspection device sets 
Z_Position(N) equal to the Start_Z location and moves camera 135 there in step 308. 
An image is acquired with camera 135 and digitized as previously described and 
stored as Image (N) in step 310. A second subroutme 312 extracts a focus value F (N) 
for Image (N) and is further described in the section for discussion for FIG. 32. A test 
is made between F(N) and the Best.Focus in step 316, such that if F(N) is greater 
than Best_Focus then Best_Focus is set to F(N) and Best_Z is set to N as shown in 
step 320 and the program flow goes onto step 322, if tiie test condition is not met m 
step 316 then the program flow skips step 320 and goes on to step 322. In step 322, a 
test is made to determine of all of the sUces have been taken as N is tested agamst the 
Number_of_Z_SUces. If N is equal to Number_of_Z_SUces then program flow goes 
onto step 324. If more sHce images are needed, then the flow goes to step 318. In 
step 318, Z_Position(N+l ) is set to Z_Position(N) + Z_Step_Size and the Z-Axis is 
moved to Z_Position(N+l ) and the program flow goes on to step 314 where N is 
mcremented by 1 (one). The program flow goes back to step 310 and completes the 
loop of step 310 to step 322 until all of the unage sUces have been taken and then 
moves onto step 324. In step 324, Best_Z is tested against its initial value, and if it 
equals its initial value (meaning no focus was found in the focus value subroutine 
312) then it is set to a default value of the Number_of_Z_SUces divided by 2 and 
Focus_Error is set to 1 (one) in step 326 and the program flow goes onto step 328. If 
Best_Z m step 324 has a value other than its initial value then program flow goes onto 
step 328 from step 324. hi step 328 a Best_hnage image is set to the image sUce at 
best focus by settmg Best_Image equal to Image(Best_Z). Also, a Best_hnage_Z 
value is set equal to Z_Position(Best_Z) and the flow goes onto step 330 which is the 
MTURN part of the subroutine and program flow returns to the main software flow. 

As illustrated in FIG 32. an image(N) focus F(N) subroutine 312 is further detailed, 
starting at the step Start 401. A pointer_to_image(N) 404 is provided in step 402. In 
step 408 the image(N) is convolved with a standard 3x3 Sobel_Horizontal filter 416 
to produce an Image(N)_H wherein horizontal edges within the image are 
emphasized. In step 410 the image(N) is convolved with a standard 3x3 
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Sobel_Vertical filter 418 to produce an Image(N)_V wherein vertical edges within 
the image are emphasized. In step 414. both the horizontal edge emphasized image 
Image(N)_H and the vertical edge emphasized image Image(N)_V are summed pixel 
by pixel, during the summing process any resulting negative pixel values are set to 
zero and any resulting pixel values that are greater than 255 are set to 255, to produce 
an image Image(N)_Focus. In step 420, a simple variance F(N) is calculated for the 
pixels within a window of mterest defined in 422 by X-Window_Center. 
Y_Window_Center. X.width. and Y_Height The resulting value of the variance is 
returned to the calling program as F(N) in step 424. ITie sobel processing and the 
variance calculation is performed with a collection of image processing software tools 
within MVTools. MVTools is available firom Coreco Imaging, US Office in 
Bedford, MA. 

Fourth Preferred Embodiment 
m the first preferred embodiment, it was disclosed how an operator could manually 
score each drop of Uquid as either "CRYSTAL" or ''NO CRYSTAL". In the fourth 
preferred embodiment, the operator is given a greater variety of options m deciding on 
how to score each drop. Table 1 shows Usting of the operator's scoring options, 
including number, text description, and the corresponding color code. Once a micro- 
weU drop has been scored a 9. the operator can further classify the crystals in a 
scoring shown in Table 2. 

TABLE 1: 



SCORE 
0 


DESCRIPTION 
clear 


DISPLAY COLOR 
White 


1 


light precipitation 


Red 


2 


heavy precipitation 


YeUow 


3 


ugly precipitation 


Blue 


4 


phase separation 


Orange 


5 


imknown 


Violet 


6 


Spherolites 


Black 


7 


Grainy precipitation 


Gray 


8 


Microcrystals 


Brown 


9 


Crystal 


Gieen 
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TABLE 2: 



SCORE 


JJJiaOxCi-r i l^iN 


9.0 


crystal (no comments) 


9.1 


needles, intergrown 


9.2 


needles, single 


9.3 


plates, intergrown 


9.4 


plates, single 


9.5 


chunks, < SOmicrons, intergrown 


9.6 


chunks, < 50 microns, single 


9.7 


chunks, > 50 microns, intergrown 


9.8 


chunks, > 50 microns, single 


9 9 gorgeous > 50 microns 



Fifth Preferred Embodiment 
In the fourth preferred embodiment, it was disclosed how an operator can manually 
score each drop of Uquid into one of 10 categories with corresponding color coding, 
and how the operator can score category 9 into further subcategories of 9.0 through 
9.9. In the fifth preferred embodiment, the inspection device automatically scores 
and classifies each drop specimen by executing computer software subroutines as 
shown in FIGS. 33, 34a, 34b, 34c, 34d, and 35a and 35b undar control of the program 
flow shown in FIG 36. The automatic classification can occur at three levels of detail, 
the first level, Type_of_Classification = 1, simply discriminates between a drop that is 
clear or not-clear (unknown), the second level, Type_of_aassification = 2, scores and 
classifies the drop into classes 0 through 9 as described in Table 1 above, and the third 
level, Type_of_Classification = 3, performs second level scoring and classification, 
plus adds an additional 10 subcategories to the CLASS 9. crystal classification, as 
detailed in Table 2 above. 



Automatic Scoring and Classification 
FIG. 36 illustrates llie main program flow 840 starting at stq) 842. The software is 
initialized with parameters, inspection Usts, and Type_of_Classification detailed in 
step 486. The flow continues onto step 844 where tiie system moves the micro-well 
of interest witiiin the selected micro-well plate under the selected camera. In step 
851. if the drop needs to be located, the flow continues onto step 849 wherein an 
image is acquired by the camera and software operates on the image and determines 
the location of the drop. Then a test is made to determine if the last micro-well in the 
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plate has been imaged, if not then the flow loops to step 844 and continues. If the last 
micro-well in plate step 853 has been imaged then the flow continues to step 856 
where the system moves to the droplet within micro-well in micro-well plate under 
the high-resolution camera. Also in step 851 if the drop had been previously located, 
then the flow would continue from step 844 directly to step 856 without the need to 
re-locate the drop. From step 856, the flow continues onto step 857 wherein a high- 
resolution image of the drop is obtained. Then the flow goes onto step 850. La step 
850, a CALL is made to subroutine that automatically scores and classifies the drop 
depending on the Type_of_Classification required. The subroutine is detailed m FIG. 
33 and starts at step 440 in FIG. 33. After the drop has been classified the subroutine 
returns to step 852 wherem the results are stored and reported The program flow 
continues to step 858 where a test is made to determine if the last drop in the selected 
plate has been processed, if so then the flow goes onto step 848 wherein a test is made 
to determine if the last plate has been processed. If not, the flow loops back to 856 
and continues. If the last plate has been processed, the flow goes onto step 854 and 
the program is done, 

FIG. 33 shows Micro-well Specimra Auto Score and Classify Subroutine 438 starting 
at step 440. Pointer_to_Best_Lnage 442 provides information to the initialization step 
444 that allows access to the image that was found to be the best focus. Plus, the 
Type_of_Classification is passed into the routine. Alternatively, pointer 442 can point 
to an image that was taken at a z height value known to be the focus of the system. 
After initialization 444 the subroutine 438 calculates in step 448 an 
average_gray_Background value normalized to allow for the variation present from 
various plate types, by using a first rectangular window defined by parameters shown 
in step 446 (X_Background ^Center, Y_Background _Center, X_Background 
^Width, and YJBackground Height) and by summing all of the gray scale values of 
the pixels defined by the window 446 and dividing by the number of pixels within 
that window. The average_grayJ)ackground is normalized for well-type differences 
by multiplying the calculated value by a Micro-well normer value also found in step 
446 and generally detennined by measuring the various micro-well plate types under 
inspection and normalizing to the 96-well standard micro-well plate. This 
average_grayJbackground 448 is calculated in a window area of the image that is 
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outside the area of the drop but generally within the well or within the bounding well 
walls. 



In step 450 an average_jTay_Classify_Window value is calculated in a similar 
manner as described above (except it is not normalized for micro-well type) using a 
second rectangular window defined by parameters shown in step 452 (i.e., X__Classify 
^Center, Y_Classify _Center, X_Classify _Width, and Y^Classify Height). This 
average__gray_classify_window value 450 is taken in a rectangular window area of the 
unage that is inside the area of the drop and defined by being a firaction between 0.98 
and 0.5 (with 0.8 preferred) of the width and height of the extemal bounding 
rectangular box &om the blob utilizing subroutine mvtJ)lob_find. The subroutine 
mvt J)lob_find defines the extent of the drop as previously discussed in the section 
^Determining the Position of the Drop of Liquid within Each Well". 

In step 454, a diagonal difference image is calculated by stepwise subtracting pixel 
values fix>m two pixel locations, defined by (x,y) and (x-fDiff_Sep, y+Di£f_Sep) 
within the Classify_window. The pixel values are separated in width and height by 
the value I>iff_sep from step 452. This is repeated over all pixels within the Classify 
window defined by step 452 using X_Classify _Center, Y_Classify _Center, 
X_Classify _Width, and Y_Classify Height. For each value calculated the absolute 
value is taken of the subtraction result and compared to a threshold value FlagJThresh 
defined in step 452. If the calculated value is greather than FlagJThresh 452 then the 
pixel is set at the first location in x,y equal to a value defined by Setl in step 452, if 
the calculated value is equal to or less than FlagJThresh, 452, then the pixel value is 
set to zero. This can be seen by the mathematical equations and flow described in 
step 454 in calcxxlating a Diff_Image_l. Typical values for DiflLSep are between 1 to 
20 pixels with 7 preferred. Typical values for Setl are between 1 and 511 with 128 
preferred. Typical values for FlagLThresh are between 5 and 50 with 25 preferred. 

In Step 456, a calculation sunilar to that performed in step 454 is performed on the 
Classify^Window except that the separation between the two pixels undergoing the 
calculation is defined by (x+Di£f_Sep, y) and (x, yfDiflF.Sq)), as is shown in the 
mathematical calculation in 456 to generate a Diff.Image_2, This calculation uses 
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Typical values for Set2 are between 1 and 511 with 



In Step 458 the Classify^Image, which is a combination of the images generated in 
step 454 and 456 is calculated as shown by the mathematical equations shown in step 
458 using definitions shown in step 452, If the x,y pixel value in either DiflF_Image_l 
or Diff_Image_2 (steps 454 and 456 respectively) has a value equal to Setl 452 then 
the pixel value is set at (x,y) m Classify_Image equal to Set2 452. Otherwise, the 
value is equal to zero(O) as shown m the mathematical equations in step 458. The 
calculations are repeated for all pixels within the window defined in 452. The 
Classify_Image is basically an image of the classify Jmage_window wherein edges 
present within the original Bestjtaiage are detected. 

In step 460, the value of number_Pixels_Set2 is set equal to the total number of pixels 
that are set equal to Set2 452 in step 458. Also, the value of Total_Pixels„in„Wmdow 
is set to the total number of pixels in the Classifyjvindow in step 458 . 

In step 462, a Score_Gray is calculated by dividing the 
Average_Gray_Classify_Window determined in step 450 by the 
Average_Gray__Background found m step 448. A Score_Flag is also calculated by 
dividing the Number_Pixels_Set2 by Total_Pixels_in_Wmdow from step 460. The 
Score_Gray and Score_Flag are normalized in this matter. 

In step 464, the values of Score^Gray, Score^Flag, and Type_of_Classification are 
passed to a classify subroutine and a classification is returned for the Classify_image, 
effectively classifying the protein crystals within the window. Details of the Classify 
subroutine 464 are provided in FIG 34a, 34b, 34c,and 34d, plus FIG 35a and 35b. 

FIG 34a shows the Classify Subroutine 464. Start of Classify subroutine is shown in 
step 468 followed by initialization 470 whereby the initial classification CLASS value 
is set to 5, representing **unknown" and the flow goes onto step 480. 

Step 480 calculates whether the drop is clear and the CLASS = 0 by a test detailed in 
step 480 using thresholds defined in step 482 (Clear.FlagLLT and Clear_Gray_GT) 
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with the following equation: if score^flag is less than Clear JFlag^LT and Sc6re_Gray 
is greater than Clear_Gray_GT then set CLASS = 0. At step 481, a test is made to see 
of the type_of_classification is equal to 1, the first classification type wherem the drop 
is classified as simply clear (0) or unknown (5) as previously discussed. If the 
type_of_classification is equal to 1 then the flow goes on to step 483 and then returns 
to FIG. 33 step 466 with die results. If the type__of_classification is not equal to 1 
then the flow goes onto step 476 for fiirther classification. 

Step 476, utilizmg threshold value parameters shown in step 478 
(Lgt_Precip_Flag_LT, Lgt_Precip_FlagLGT, Lgt_Precip_Gray_GT), assigns the 
value of 1 to CLASS indicating that Light Precipitation is present in the 
Classify_Image. Step 476 utilizes the mathematical equation which states if 
Score__Flag is less than Lgt_Precip_Flagj:.T and Score_Flag is greater than 
LgtJ*recipJFlag^GT and Score_Gray is greater than Lgt_Precip_Gray_GT then set 
CLASS to value 1. 

Step 484 calculates heavy precipation by using thresholds detailed in step 486 
(Heavy„Precip_Flag^LT, Heavy_Precip_Flag_GT, Heavy_Precip_Gray_LT) with 
the following equation: if score_flag is less than Heavy_Precip_FlagLLT and 
score_flag is greater than Heavy_Precip_Flag_GT and score_gray is less than 
HeavyJPrecip_Gray_LT then set CLASS = 2. 

Step 488 calculates ugly precipation by using thresholds detailed in step 490, 
Ugly_Precip_FlagL_LT, Ugly_Precip„FlagLGT, Ugly„Precip_Gray„LT, with the 
following equation: if score_flag is less than Ugly„Precip„Flag^LT and score^flag is 
greater than Ugly_Precip__Flag_GT and score^gray is less than 
Ugly_Precip_Gray_LT then set CLASS = 3. 

Step 492 continues the classification process in FIG 34b. 

In FIG 34b, the contmuation of the classification process 700 is shown continuing in 
step 702. Step 704 calculates micro^crystals by using thresholds detailed in step 703 
(Micro_Cry_FlagLGT and Micro_Cry_^Gray_LT) with the following equation: if 



20 



wo 03/034030 



PCT/US02/33150 



score__flag is greater than Micro_CryJFlag^GT and score _gray is less than 
Micro_Qx_Gray_LT, then set CLASS = 8. 

Step 704 calculates crystals by using thresholds detailed in step 705 
(Crystal„Flag^LT, Crystal^Flag^GT, and Crystal_Gray_GT) with the following 
equation: if score_flag is less than Crystal_Flagj:,T and score^flag is greater than 
Crystal_Flag^GT and score^gray is greater than Crystal__Gray_GT then set CLASS = 
9. 

Step 706 calculates Graiay precipitation by using thresholds detailed in step 707 
(Grainy_Flag_GT, and Grainy_Gray_GT) with the following equation: if CLASS = 
8 and score_flag is greater than Grainy_Flag_GT and score _gray is greater than 
Grainy_Gray_GT, then set CLASS = 7. 

Step 708 continues the classification process onto FIG 34c as 725. 
Step 726 continues firom step 708 of FIG 34b. 

In FIG 34c, Step 727 calculates and generates a first set of additional image features 
for further use in classification taking as ii5)ut 726 (X_Classify ^Center, Y^Classify 
^Center, X_Classify _Width, Y„Classify Height, PointerJo_classify_iniage, and 
mvt_blob__analysis_Parains). In step 727, MVT_Tools_Blob_Analysis with pointer to 
window in Classify_finage is called. Step 727 gets the num_found of blobs and, for 
each blob found, step 727 gets its aiea(m), height(m), width(m), perimeter(m), and 
location of each as location_X(in) ands location_Y(m). These values are recorded. 
These calculations are performed on the image called Classify_Iinage, which is the 
image that was formed in FIG 33 as step 458. 

In step 728, if num_found is not greater than zero(0) the subroutine goes to step 738. 
However, if any blobs are found then further analysis is started in step 729 by setting 
m=l, num_spherolite = 0, and spherolite(m) = 0. In step 730, the following 
blob^ratios are calculated: Ciicle_Like_HW(m) = height(m)/Width(m) and 
Circle_Like_AHW(m) = area(m)/(height(m)*width(m)). For blobs that are circular, 
Circle_Like_HW will be around a value of one (1). If blobs get elongated then the 
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value will be other than one. Circle_Like_AHW for circular blobs has a value 
around 0.785. For square-like blobs the value will be closer to one (1). The program 
flow goes onto step 732. 

Step 732 determines whether to classify a drop as having a spherolite by utili2mg the 
parameters found in 734 (Circle_Like_HW Jower, Circle_Lik:e_HW_Upper, 
Circle_Like_AHW_Lower, and Circle„Like_AHW„Upper). The following equation 
is used:: if CLASS = 8 or 9 and Cuxle_Like_HW(m) is less than 
Circle_LikeJIWJJpper and Circle_LikeJBW(m) is greater than 
Ciicle_Like_HWJower, and Ckcle_Like_AHW(m) is less than 
Circle_Like_AHW„Upper and Circle_Like_AHW(m) is greater then 
CiTCle_Like_AHW_Lower, thennum_spherolite = num_sperolite -fl. An increment 
is done by calculating m = m+1 and setting spherolite(m) = 1 to show one has been 
found at m. 

Step 736 tests whether all of the found blobs have been classified by testing m against 
num_found and, if equal, the subroutine goes onto step 738. If not the program loops 
back to step 730 and flows through as above. 

Step 738 goes onto FIG 34d in the classification subroutine as 740. 

FIG 34d continues from FIG 34c at 742 and goes to 744 where a second set of 
additional features is calculated. These features are generated with the origmal 
Bestjmage generated in step 328 (FIG. 31) as the image of the best focus. For these 
additional features, the results are used from the blob analysis performed in FIG. 34c 
on the classify_image. Such features as shown in parameters 746 as num_found, 
area(m), height(m), width(m), perimeter(m), location_X(m), location_Y(m), 
num_spherolite, and spherolite(m). Plus it uses the average_gray background from 
FIG. 33 step 448, and the pointerJo_Best_Ihnage in FIG. 31 step 328. In step 744 
* W and numjphase_sep are set equal to zero. 

In step 748, if the number of spherolites previously found (num_spherolite) is not 
greater than zero (0), then the flow goes to step 756 and the remaining steps shown in 
FIG 34d are bypassed. But if the num^spherolite is greater than zero (0), then the 
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flow goes onto step 749 wherein m is incremented by 1. Then the flow goes on to 
step 751 to test whether to go to step 756 or to step 753 depending on comparing '"m" 
to the num^found. In step 753, the value of spherolite(m) foxind in FIG. 34c step 732 
is tested to see if any were classified as spherolite. If not, then the flow loops back to 
step 749. If spherolite(m) is equal to one (1) then step 750 is executed. 

In step 750, the mformation location_X(m), location_Y(m), one half of heigjit(m) 
and one half of width(m) is used to calculate the average gray value within this 
reduced image inside each spherolite. The Best__Image pixel data obtained firom the 
previous blob analysis is utilized and phase_sep_Gray(m) is set equal to this value. 
Phase_sep_Gray(m) is then normaUzed by dividing it by the 
Average_Gray_Background The program flow goes onto step 752, wherein 
phase_sep_Gray(m) is tested to see if it is greater than a parameter 
phase_sep_Gray_GT from pairameter input step 754. If true, then CLASS is set equal 
to four(4) and the program loops up to step 749. 

In step 751 the automatic program has classified the microdrop into one of the 10 
primary classes, 0 through 9, detailed previously m Table 1. Then, a test is made to 
see if the type _of_classification is equal to 2. If so, then the flow goes onto step 755 
wherein the flow is returned to FIG. 33. In step 466, a general classification is 
complete. If further classification into subcategories is required, then the flow goes 
onto step 758. 

In step 758, the program flow goes onto FIG. 35a step 760. 

In FIG. 35a, a crystal classification subroutme 762 fiarther classifies any CLASS 9 
crystal image into to additional subclasses, 9.2, 9.4, 9.6, 9.8, or 9.9, as previously 
discussed in Table 2. The subroutme begins at step 760 and goes onto step 764. In 
step 764, the blob counter m is set to zero. A test is conducted to see if the CLASS is 
equal to 9 and, if not, the flow goes to step 778 and returns to FIG. 33 step 466. Step 
764 uses input values as shown in step 766 (Average_Gray_Backgroun4 
Pointer_To_Best_Image, num^found, area(m), heigiht(m), width(m), perimeter(m), 
locationJX(m), location_Y(m), num^spherolite, and spherolite(m)), as previously 
described. If the CLASS is equal to 9 another test is made to see if one or more than 
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one blobs were previously foim(L K only one blob was previously found then the 
flow goes onto step 768. If more then one is found, the flow goes onto step 769 in 
FIG 35b. In step 768, a height to width ratio is calculated and compared to thresholds 
representing needle-like characteristics in step 770. If the test conditions are met, 
then the CLASS is set to 9.2. If not, the flow goes onto step 771. In step 771, the 
height to width ratio is compared to thresholds representing plate-like characteristics 
m step 772. If the conditions of the test are met, the CLASS is changed to 9.4 and the 
flow goes onto step 773. If not, no further classification is performed and the flow 
goes onto step 778 and the subroutme returns to FIG. 33 step 466. Li step 773, a 
normalized average gray value, Chunk_Gray(m), within the blob is calculated and the 
flow goes onto step 774. In step 774, if Chunk _gray(m) is less than a threshold 
Chunck_Gray_LT (from step 775), then CLASS is set to 9.6 and the flow goes onto 
step 776. In step 776, if Chunk _gray(m) is greater than a threshold Chunk_^50_GT, 
(from step 777), then CLASS is set to 9.8 and the flow goes onto step 780. Instep 
780 if area(m) is greater than threshold gorgeous_GT (from step 779), then CLASS = 
9,9 and the flow goes onto step 778. Then the subroutine returns to FIG. 33 step 466. 

FIG 35b shows flowchart 781 illustratmg the further classification of a CLASS 9 
image havmg multiple blobs within the unage into additional subclasses, 9.1, 9.3, 9.5, 
or 9,7, as previously discussed in Table 2. The subroutme begins at step 786 from 
step 769 in FIG. 35a and goes onto step 782. In step 782, a loop begins at m equals 0 
for each blob m using input values from step 784 (Average_Gray__Background, 
Pointer_To_Best_Image, num_found, area(m), height(m), width(m), perimeter(m), 
location_X(m), location_Y(m), num_spherolite, and spherolite(m)) as previously 
described. The flow goes onto step 788 where m is incremented and a height to width 
ratio is calculated and compared to thresholds representing needle-like characteristics 
in step 790. If the test conditions are met, then the CLASS is set to 9.1. If not, the 
ilow goes onto step 794, hi step 794, the height to width ratio is compared to 
thresholds representing plate-like characteristics in step 796. If the conditions of the 
test are met, the CLASS is changed to 9.3. If not, no further classification is 
performed and the flow goes onto step 812 to determine if all of the blobs have been 
tested. In step 798, a normalized average gray value (Chunk_Gray(m) as described 
before) within the blob is calculated and the flow goes onto step 802. In step 802, if 
Chunk _gray(m) is less than a threshold Chunk^Gray^LT (from step 804) then 
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CLASS is set to 9.5 and the flow goes onto step 806. In step 806, if Chniik^ay(m) 
is greater than a threshold Chunck„50_GT (from step 808) then CLASS is set to 9.7 
and ttie flow goes onto step 812. In step 812, if all of the blobs have been tested, flie 
flow goes onto step 810 and the subroutine returns to FIG. 33 step 466, If not aU 
tested, then the flow loops back to step 788 for tiie next blob and the process loops 
until complete. 

Typical parameter and threshold values for use in classification in FIGS. 33, 34a, 34b, 
34c, 34d, and 35a and 35b are given in Table 3 along with the preferred value. These 
values serve only as a guide, and other values may be used when circumstances 
justify. For example, different lighting conditions, variations in the tranq)arency of 
micro-well plates, variations in the formulations of the protem growmg media and 
drop, values may be used as well. One skilled in the art may adjust the parameter and 
threshold values to tune in the classification results specific to their setup. 



Table 3 Typical and preferred values for threshold and Classification parameters 



step 


Name 


lower 
value 


upper 
value 


Preferred 
value 


452 


Di£f_Sq) 


1 


20 


7 


452 


Setl 


1 


511 


128 


452 


Set2 


1 


511 


200 


452 


Fla&_Thresh 


5 


50 


25 


478 


Lgt_Precip_Flag^LT 


0.0001 


0.010 


0.002 


478 


Lgt_Precip_Flag^GT 


0.0 


0.001 


0.00001 


478 


LgtJPrecip_Gray_GT 


0.8 


1.2 


1.020 


482 


Clear_Flag^LT 


0.000001 


0.01 


0.0007 


482 


Clear_Qray_GT 


0.8 


1.2 


1.020 


486 


Heavy_Precip_Flag_LT 


0.0001 


0.050 


0.020 


486 


HeavyJPrecipJFlag^GT 


0.0 


0.005 


0.001 


486 


Heavy_Precip_Gray_LT 


0.7 


1.2 


0.95 


490 


UgIy^Precip_Flag_LT 


0.0001 


0.050 


0.020 
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490 


Ugly_Precip_Flag^GT 


A A 

0.0 


0.0U5 


A AA1 

0.001 


490 


UglyJPrecip_Grax_LT 


A C 

0.5 


1.2 


A 7 

0./ 


703 


Micro_CryJFlagLGT 


A AAA1 

0-0001 


A A€ 
0.U5 


A A7A 

0.020 


703 


Micro_Cry_Gray_LT 


A A 

0,9 


1 OA 

1.20 


1 Al A 
1.010 


705 


Crystal_flag_LT 


0.1 


0.5 


A OA 

0.30 


705 


Crystal_Fla&_GT 


A AAI 

0.001 


A AC 

0.05 


A Al 

0.01 


705 


Crystal_Gray_GT 


A Q 

0.8 


1.2 


1 AAAA 

i.ooyy 


707 


Grainy_FlagL.GT 


AA1 

.001 


.5 


A 1 77 
0.1 / / 


707 


Grainy_Gray_GT 


A O 


1 7 

1.2 




734 


Circle_L±e_HW_lower 




1 AA 
l.UU 


n o 


734 


Circle_Like_IlW_Upper 


l.UU 


7 A 
2.0 


1 1 
1.1 


734 


Circle_Like_AHW_lower 


A 1 

0.1 


7 A 


A 7Q 
0. /o 


734 


Cuxle_Like_AHW_Upper 


.785 


O A 

2.0 


A Q7 


754 


phase_sep_Gray_GT 


A A 

0.9 


1 7 

1.2 


l.OD 


770, 790 


needle JEIW_GT 


2 


OA 

20 




770, 790 


needle_HW_LT 


A C 

0.5 


A AC 

O.OD 


A 7 

0.2 


772, 796 


1 J. TTTTT X^Tl 

plate J[IW_GT 


1 1 

1.1 . 


7 A 

2.0 


A 


772, 796 


■f TTTTT T rri 

plate JIW„LT 


A A 

0.9 


1.1 


1 A^ 
1.U3 






0 8 


1.2 


1.01 


777, 808 


Chunk_50_GT 


70 


200 


100 


779 


Gorgeous_GT 


7000 


2000 


1000 













Prototype 

Applicants have designed, built and tested a working prototype of the present 
invention. 

FIG. 30 shows the major components of the Applicant's prototype. Proteomic crystal 
verification and inspection system 100 has three axis of linear motion. Lmear actuator 
115 is preferably linear actuator model # 802-0763D available firom Dynamic 
Solutions of Irvine CA., with 600 mm of travel driven by an ^closed 0.5 mm pitch 
ballscrew. Linear actuator 1 15 is driven by an intelligent self-contained servo motor 
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110 model # SM2320 SQ available from Animatics Corp. of Santa Clara, CA., with 
38 oz-in of available torque. Servo motor 110 communicates with a Windows Based 
computer 105 through a serial connection routed through a central control unit 190. 

Lmear actuator 115 has stationary part 116 fixed to a granite table top 170, Motor 
1 10 moves moving part 1 17 along the x-axis. Granite top 170 is supported by a fi:ame 
180. Frame 180 has casters and adjustable legs. Plate 127 is attached to moving part 
117. At each end of plate 127, a spacing block 128 spaces fixture plate 129 &om 
plate 127. At each of its ends, fixture plate 129 is supported by spacing block 128. 
Fixture plate 129 provides for the mounting, removal, and positioning of multiple 
micro-well plates 125A - 125F. Preferably micro-well plates 125A - 125F are agar 
plates, micro-titer well plates of 96, 384, 1536 wells, or other sample plates. 

Support 191 is positioned adjacent to fixture plate 129 and contains a light source. In 
the preferred embodiment, the light source is model # A08925 fiber-optic back light 
available from Aegis Electronics of Carlsbad, CA. 

In the preferred embodiment, linear actuator 150 is model # 802-1384A available 
from Dynamic Solutions of Irvine CA., with 350 mm of travel driven by an enclosed 
0.5 nam pitch ballscrew. Linear actuator 150 has stationary part 153 horizontally 
bridged above linear actuator 115 and siq>ported by piUars 152. Moving base 154 
provides a mounting base for components of Unear actuator 160. In a preferred 
embodiment, linear actuator 150 is driven by driven by an intelligent self-contained 
servo motor 120 identical to motor 110. Servo motor 120 communicates with 
Windows Based coniputer 105 throu^ a serial connection routed through central 
control tinit 190. 

In a preferred embodiment, linear actuator 160 is model # 802-0756D available firom 
Dynamic Solutions of Irvine CA., with 100 mm of travel driven by an enclosed 0.5 
mm pitch ballscrew. Linear actuator 160 has with a stationary part 161 mounted 
perpendicular to and fixed to moving base 154 of linear actuator 150. Linear actuator 
160 is driven by intelligent self-contained servo motor 130. Servo motor 130 is 
preferably identical to motor 110. Servo motor 130 communicates with tiie Windows 
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Based computer 105 through a serial connection routed through the central control 
unit 190. 

Moving plate 162 provides a mounting base for camera 155. In a preferred 
embodiment camera 155 is a high-resolution monochrome megapixel CCD camera 
model # CVMl from JAI America INC. of Laguna Hills, CA. Camera 155 has lens 
165. Preferably, lens 165 has a 0.75x to 3x zoom capability and is model # NT52-571 
available from Edmund Industrial Optical of Barrington, NJ. Moving plate 162 also 
provides a mounting base for camera 135. Preferably, camera 135 is a high-resolution 
monochrome meg^ixel CCD camera model # CVMl from JAI America INC. of 
Laguna Hills, CA. Camera 135 has lens 145. Preferably, lens 145 has a 2.5x to lOx 
zoom c^ability and is model # NT54-396 available from Edmund Industrial Optical 
of Barrington, NJ. 

Moving plate 162 also provides a mounting base for a zoom lens motor 192, an 
intelligent self-contained servo motor model # SM2315D available from Animatics 
Corp. of Santa Clara, CA., with 20 oz-in of available torque. The servo motor 
communicates with the Windows Based computer 105 through a serial connection 
routed through the motion control box 190. The zoom lens motor operates the zoom 
lens 145 through a conventional belt drive. 

A bar-code reader 175 is mounted adjacent linear actuator 115 and is attached to 
support 172 fixed on granite top 170. Bar-code reader 175 is positioned to view a 
bar-code identity label attached to a micro-well plate when the micro-well plate is 
positioned under linear actuator 150. Preferably, bar-code reader 175 is model # 
BL60 1 available from Keyence Corporation of America of Newark, NJ. 

A plate sensor transmitter/receiver 186 is also fixed to support 172 and is aligned to 
sense whenever a micro-well plate 125 breaks a beam of light emitted by 
transmitter/receiver 186 and reflected.by a reflector 188. Reflector 188 is mounted on 
support 191 on the opposite side of the linear actuator 115. Plate sensor 
transmitter/receiver 186 and reflector 188 are preferably model # E3T-SR13 available 
from Westem Switch Controls of Santa Ana, CA, 
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Block Diagram Showing Connectivity of the Prototype 
FIG. 6 shows a block diagram illustrating the connectivity of Applicant's prototype. 
Linear actuator motors 110, 120 and 130, and zoom motor 192 receive DC power 
from 48 Volt DC power supply 680 through an electrical connection- Linear actuator 
motors 110, 120 and 130, and zoom motor 192 motors conmiunicate with Windows- 
based computer 105 through common serial line 633. Bar-code reader 175 
communicates with computer 105 tiirough a communications line and plate sensor 
186 communicates with the computer 105 through a communications line. Monitor 
620 displays information to the operator transmitted from computer 105. Cameras 
135 and 155 communicate with frame grabber 630 through communication lines. 
Frame grabber 630 is installed within computer 105 and is preferably PCVision from 
Coreco Imaging US Office in Bedford, MA.. Frame grabber 630 digitizes the hnage 
from the camera to form the digitized image data array withm computer 105. A 4- 
port Ethernet hub 640 provides for connectivity between computer 105, central 
control unit 190, and an external Ethernet 670. By providing for connectivity to the 
Ethernet, computer network communications are possible. The central control unit 
1 90 controls light source 1 94 through an analog control Une. Central control unit 1 90 
receives 24 volt DC power from 24 volt DC power supply 660. Emergency stop 
button and switch (e-stop) 650 is connected to central control unit 190. 

Experimental Results 

Fifty-three test images were obtained from the system and were both automatically 
classified by the system and were manually classified by four scientists. Table 4 
shows the correlation percratage between the various scientists and the automatic 
classification provided by the system. 

Table 4 





Sam 


Mary 


Susan 


Red 


AUTO 


Sam 


100% 










Mary 


98% 


100% 








Susan 


93% 


97% 


100% 






Fred 


89% 


93% 


96% 


100% 




AUTO 


95% 


93% 


86% 


81% 


100% 
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Utilization of Color 

In another preferred embodiment, the present invention is configured to record color 
images. It is desirable to be able to analyze color unages in that certain precipitation 
products in the protein crystallization process have distinctive colors and a 
crystallographer or automated image analysis algorithm may use the color information 
to help discriminate crystallization results. 

True Color Picture 

FIG, 37 shows a side view of micro-well plate 125A positioned on fixture plate 129. 
Support 191 with embedded light source 194 is positioned to the side of fixture plate 
129. Light fix)m light guide 195 is directed upward through cutout 131. Light guide 
195 is positioned between fixture plate 129 and plate 127 such that both plates can 
move around the light guide 195 without interference. 

Linear polarized filter 352 is rotationally mounted above light guide 195 such that 
hght from Ught guide 195 can be polarized linearly at a programmable angle before it 
transits through micro-well plate 125A. Polarizer drive belt 356 (top view shown in 
FIG. 38) rotates polarized filter 352 about a vertical axis. Polarization drive belt 356 
is driven by motor 358. Motor 358 is controlled by CCU 190 (FIG. 40). Second 
filter 354 (top view shown in FIG. 39) is positioned above micro-well plate 125A 
such that light transiting through micro-well plate 125A goes through second filter 
354 before it goes into the camera zoom lens 145. Second filters 354 are mounted on 
filter wheel q)paratus 355. Filter wheel apparatus 355 rotates the operator selected 
second filter 354 into position under the zoom lens 145. The selected second filter 354 
is preferably either a red, green or blue dichroic filter. Preferably, individual images 
taken through the red, green and blue second filters are combined to form a true color 
image. 

False Color Image 

In addition to the true color unages that may be formed using red, green, and blue 
filters 354, a false color (also called a pseudo color) image may be formed by taking 
three mdividual unages using linear polarized filter 352 at three different polarization 
angles with respect to a second filter. In this preferred embodiment, a linear polarized 
filter is substituted for the dichroic second filters 354 discussed above. For example, 
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the polarized axis may be at 90 degrees to each other, and at plus and minus 45 
degrees to each other, The three images are then called red, green, and blue and a 
false color image is produced. If the crystal exhibits any polarization rotation effects, 
then a very colorful image results. This pseudo color unage is useful in detecting very 
small and fine crystals firom the fanage background material. Other polarizmg angles 
may be selected as well. 

In the preferred embodiment, light guide 195 is model # A08925 fiber-optic backlight 
available fiom Aegis Electronics of Carlsbad. CA. Light source 194 is a Dolan- 
Jenner Model-PI^900 available firom Edmund ludustrial Optics, Barrington, NJ. First 
polarized filter 352, second filter 354 (including the linear polarized filters, and the 
dichroic. red, green, blue filters) are available firom Edmund Industrial Optics, 
Barrington, NJ. Filter wheel 355 is model FWl available &om Integrated Scientific 
Imaging Systems, Inc of Santa Barbara, CA. 



Although the above-preferred embodiments have been described with specificity, 
persons skilled in this art will recognize that many changes to the specific 
embodiments disclosed above could be made without departing firom the spirit of the 
invention. For example, although the above preferred embodiments specifically 
describe an indexing device in which linear actuators 115, 150, and 160 operate in 
conjunction to sequentially position protein crystals under cameras 155 and 135, there 
are a variety of other types of robotic indexing devices that could also be utilized for 
the same purpose. For example, an indexing device could be built in which the 
plurality of micro-well plates are kept m a stationary position. The camera lens would 
be attached to an indexing device that is preferably capable of unrestricted movement 
in the horizontal plane. The camera lens would be moved sequentially firom micro- 
well to micro-well in the horizontal plane. Once in position over a micro-well, the 
lens could be raised or lowered in the vertical direction to achieve proper zoom and 
focus. In another embodiment, an indexiag device could be built in which cameras 
155 and 135 are kept stationary with respect to horizontal movement In this 
embodiment, the plurality of micro-well plates would be preferably placed on a 
positioning platform that is capable of unrestricted movement in the horizontal plana 
In this fashion, the positioning platform could be moved so that each micro-well is 
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sequentially positioned underneath the ^propriate camera. As witti the previous 
embodiment, once in position over a micro-well, the lens could be raised or lowered 
in the vertical direction to achieve proper zoom and focus. Also, although the first 
preferred embodiment discussed inspecting crystals grown by the hanging drop 
method, other crystals grown utilizing other methods could be inspected with equal 
effectiveness. For example, FIG. 12 shows protem crystal growth as a result of 
aqueous drop in oil protein crystallization. Cameras 135 and 155 focus on the crystals 
in drop 362. Also, although the above preferred anbodiments discussed in detail how 
the present invention is utilized for inspecting protein crystals inside drops of liquid, 
the present invention could also be utilized to uispect other types of microscopic 
specimens. For example, the present invention could be utilized to inspect typical 
micro-well micro titer plate reactions wherein the quality of the reaction can be 
judged by the amount and wavelength of fluorescence emitted by the specimen by 
configuring the system with appropriate light sources, filters, and sensitive cam^as as 
is typical for fluorescence detection. . Also, although the above preferred 
embodiments disclosed the utilization of two cameras 135 and 155, it would also be 
possible to have just one camera that is capable of zooming out so that it can focus on 
tihe entire well and zooming in so that it can focus on the drop of liquid containing the 
crystal. In additioia, although an area CCD camera is shown, a linear CCD camera 
combined with moving of the micro-well plate would also work in the present 
invention. Also, in another preferred embodiment the detents 510 and 520 can be 
simply spring loaded and not controlled by the computer 105. Also, the invention is 
taught with a light panel light source, and or a bulb, likewise LED light sources and 
laser light sources coxild also be used with the present inventioa Although the system 
is shown that only moves the micro-well plates in one axis and the camera in the other 
two axes, the rnv^ation could likewise be practiced with either the micro-wells 
moving in two orthogonal axes (such as X and Y) while the camera moves only in the 
Z-axis or the motion of all three axes be done with the camera system, wherein the 
micro-well plates are stationary and the system moves above them. These other 
variations of system design could also require rearrangement of the light source or 
multiple light sources. Also, other filter types may be substituted for second filter 
354. For example, a linearly polarized filter would be very effective. Also, althougih 
the above preferred embodiments disclosed specific types of cameras 135 and 155, 
other CCD cameras may be used m the present invention witii less resolution or with 
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greater resolution and still practice the present invention. For example, cameras of 
2,000 by 2,000 pixels and even 4,000 by 4000 pixels are commercially available &om 
several vendors. When digitizing these alternative cameras, the digitized image 
would have the corresponding resolution of the camera. Also, one may practice this 
invention and digitize to greater gray-scale accuracy than 8-bit and gain advantage if 
the camera supports the greater bit depth, for example if the camera were cooled to 
reduce image noise. Therefore, the attached claims and their legal equivalents should 
determine the scope of the invention. 
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We claim: 

1) A device for inspecting and classifying a plurality of microscopic crystals, 
comprising: 

A) at least one camera, 

B) an indexing device for sequentially placing said microscopic crystals in 
camera-view of said at least one camera, and 

C) at least one computer programmed to control said indexing device and said at 
least one camera, wherein said at least one computer is programmed to receive 
from said at least one camera images of said plurality of microscopic crystals, 
wherein said at least one compute is programmed to classify said plurality of 
microscopic crystals. 

2) The device for inspecting and classifying a plui^ty of microscopic crystals as in 
Claim 1, further comprising a computer monitor, wherein an operator interfacing 
with said at least one conq)uter manually inputs a score to classify said pluraUty of 
microscopic crystals after observing said plurality of microscopic crystals on said 
computer monitor. 

3) The device for inspecting and classifying a pluraUty of microscopic crystals as in 
Claim 1, wherein said at least one computer automatically classifies said plurahty 
of microscopic crystals after receiving said images. 

4) The device for inspecting and classifying a plurality of microscopic crystals as in 
Claim 1, wherein said at least one camera is a first camera and a second camera, 
wherein said first camera is set at a constant zoom to cover a large field of view, 
wherein said second camera comprises a motorized zoom lens controllable by said 
at least one computer. 

5) The device for in£5)ectmg and classifying a plurality of microscopic crystals as in 
Claim 1, wherem each of said pluraUty of microscopic crystals are contained 
within a drop of Uquid, wherein each drop of Uquid is contained within a well of a 
micro-well plate. 
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6) The device for inspectmg and classifyin| a plurality of microscopic crystals as in 
Claim 1, wherein said plurality of microscopic crystals are a plurality of protein 
crystals inside the wells of at least one micro-well plate. 

7) The device for inspecting and classifying a plurality of microscopic crystals as in 
Claim 1 , wherein said indexing device comprises: 

A. a jSrst linear actuator for horizontally positioning said positioning station, 

B. a second linear actuator for horizontally positioning said at least one camera, 
and 

C. a third linear actuator for vertically positioning said at least one camera. 

8) The device for inspecting and classifying a pluraUty of microscopic crystals as in 
Claim 7, wherein said at least one camera is a first camera and a second camera, 
wherein said first camera is set at a constant zoom to cover a large field of view, 
wherein said second camera comprises a motorized zoom lens controllable by said 
at least one computer, 

9) The device for inspecting and classifying a plurality of microscopic crystals as in 
Claim 1, further comprising a bar-code reader and a micro-well plate sensmg 
device, 

10) The device for inspecting and classifying a pluraUty of microscopic crystals as in 
Claim 1, wherein said at least one computer comprises a jframe grabber for 
receiving images of said plurality of microscopic crystals from said at least one 
camera. 

11) The device for inspecting and classifying a plurality of microscopic crystals as in 
Claim 1, wherein said at least one computer is comiected to a computer network. 

12) The device for inspecting and classifying a pluraUty of microscopic crystals as in 
Claim 11, wherem said at least one computer network is the Ethernet. 

13) A method for inspecting and classifying a pluraUty of microscopic crystals, 
comprising the steps of: 
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A) sequentially placing said plurality of microscopic crystals in camera-view of at 
least one camera utilizing an indexing device, 

B) taking ttie images of said plurality of microscopic crystals with said at least 
one cam^a, 

C) transferring said images to at least one computer, 

D) receiving said images at said at least one computer, and 

E) classifying said images at said, at least one computer, 

wherein said at least one computer is programmed to control said at least one 
indexing device and said at least one camera. 

14) The method for inspecting and classifying a plurality of microscopic crystals as in 
Claim 13, wherein said step of classifying said images at said at least one 
computer is done manuaUy, wherein an operator interfecing with said at least one 
compute and viewing a computer monitor manually inputs into said at least one 
computer a score to classify said plurality of microscopic crystals after observing 
said plurality of microscopic crystals on said computer monitor. 

15) The method for injecting and classifying a plurality of microscopic crystals as in 
aaim 13, wherein said step of classifying said images at said at least one 
conq)uter is done automatically at said at least one conq)uter after receiving said 
images at said at least one computer. 

16) The method for inspecting and classifying a plurality of microscopic crystals as in 
Claim 13, wherein said at least one camera is a first camera and a second camera, 
wherein said first camaa is set at a constant zoom to covct a large field of view, 
wherein said second camera comprises a motorized zoom lens conti»llable by said 
at least one computer. 

17) The method for inspecting and classifying a plurality of microscopic crystals as in 
Claim 13, wherein each of said plurality of microscopic crystals are contained 
witiiin a drop of liquid, wherein said each drop of liquid is contained witiiin a well 
of a micro-well plate. 
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18) The method for inspecting and classifying a plurality of microscopic crystals as in 
Claim 13, wherein said plurality of microscopic crystals are a plurality of protein 
crystals inside the wells of at least one micro-well plate, 

19) The method for inspecting and classifying a plurality of microscopic crystals as in 
Claim 13, wherein said indexing device comprises: 

A. a j5rst linear actuator for horizontally positioning said positioning station, 

B. a second Unear actuator for horizontally positiorung said at least one camera, 
and 

C. a third linear actuator for vertically positioning said at least one camera. 

20) The method for inspecting and classifying a plurality of microscopic crystals as m 
Claim 19, wherein said at least one camera is a first camera and a second camera, 
wherein said first camera is set at a constant zoom to cover a large field of view, 
wherein said second camera comprises a motorized zoom lens controllable by said 
at least one computer. 

21) The method for inspecting and classifying a plurality of microscopic crystals as m 
Claim 13, wherein said indexing device fiirther comprises a bar-code reader and a 
micro-well plate sensing device. 

22) The device for inspecting and classifying a plurality of microscopic crystals as in 
Claim 13, wherein said at least one computer comprises a firame grabber for 
receiving images of said pluraUty of microscopic crystals firom said at least one 
camera. 

23) The method for inspecting and classifying a plurality of microscopic crystals as in 
Claim 13, wherein said at least one computer is connected to a computer network. 

24) The method for inspecting and classifying a plurality of microscopic crystals as in 
Claim 23, wherein said at least one computer network is the Ethernet 
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25) The method for inspecting and classifying a plurality of microscopic crystals as in 
Claim 13, wherein said images of said plurality of microscopic crystals are color 
images. 

26) The method for inspecting and classifying a plurality of microscopic crystals as in 
Claim 25, wherein said color images are true color images wherein said step of 
taking the images of said plurality of microscopic crystals comprises the sub steps 
of: 

A) viewing each of said pluraUty of microscopic crystals through a first linear 
polarized filter and a second colored filter, wherein the color of said second 
colored filter is altered between red, green and blue, to obtain a first image, a 
second image and a third image, and 

B) combining said first image, said second image and said third image to obtain 
said true colored images. 

27) The method for inspecting and classifying a plurality of microscopic crystals as in 
Claim 25, wherein said color images are false color images wherein said step of 
taking the images of said pluraUty of microscopic crystals comprises the sub steps 
of viewing each of said plurality of microscopic crystals through a first linear 
polarized filter and a second linear polarized filter to obtain said false color 
images. 

28) The method for inspecting and classifying a plurality of microscopic crystals as in 
Claim 13, wherein said step of classifying said images at said at least one 
computer comprises the sub steps of: 

A) comparing said images to a database stored on said at least one computer, and 

B) assigning a classification based on said comparison. 

29) The method for uispecting and classifying a plurality of microscopic crystals as in 
Claim 28, wherein said sub step of assigning a classification based on said 
comparison comprises the sub steps of: 

A) making a determination as to whether a microscopic crystal exists, 

B) ma kin g a determination as to what is present if said microscopic crystal does 
not exist, and 
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C) making a determination as to the qualities of said microscopic crystal if said 
inicroscopic crystal does exist 

30) A device for inspecting and classifying a plurality of microscopic crystals, 
comprising: 

A) at least one camera means, 

B) an indexing means for sequentially placing said microscopic crystals in 
camera-view of said at least one camera means, and 

C) at least one computer means programmed to control said indexing means and 
said at least one camera means, wherein said at least one computer means is 
programmed to receive from said at least one camera means images of said 
plurality of microscopic crystals, wherein said at least one computer means is 
programmed to classify said plurality of microscopic crystals. 

31) The device for inspecting and classifying a plurality of microscopic crystals as in 
Claim 30, further comprising a computer monitor means, wherein an operator 
interfacing with said at least one computer means manually inputs a score to 
classify said plurality of microscopic crystals after observing said plurality of 
microscopic crystals on said computer monitor means. 

32) The device for inspecting and classifying a plurality of microscopic crystals as in 
Claim 30, wherein said at least one computer means automatically classifies said 
plurality of microscopic crystals after receiving said images. 

33) The device for inspecting and classifying a plurality of microscopic crystals as m 
Claim 30, wherein said at least one camera means is a first camera and a second 
camera, wherein said first camera is set at a constant zoom to cover a large field of 
view, wherein said second camera comprises a motorized zoom lens controllable 
by said at least one computer means. 

34) The device for inspecting and classifying a plurality of microscopic crystals as in 
Claim 30, wherein each of said plurality of microscopic crystals are contained 
within a drop of liquid, wherein each drop of liquid is contained within a well of a 
micro-well plate. 
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35) The device for inspecting and classifying a plurality of microscopic crystals as in 
Claim 30, wherein said plurality of microscopic crystals are a plurality of protein 
crystals inside the wells of at least one micro-well plate. 

36) The device for inspecting and classifying a plurality of microscopic crystals as in 
Claim 30, wherein said indexing means comprises: 

A. a first linear actuator means for horizontally positioning said positiomng 
station, 

B. a second linear actuator means for horizontally positioning said at least one 
camera means, and 

C. a third linear actuator means for vertically positioning said at least one camera 
means. 

37) The device for inspecting and classifying a plurahty of microscopic crystals as in 
Claim 36, wherein said at least one camera means is a first camera and a second 
camera, wherein said first camera is set at a constant zoom to cover a large field of 
view, wherein said second camera comprises a motorized zoom lens controllable 
by said at least one computer means. 

38) The device for inspecting and classifying a plurality of microscopic crystals as in 
Claim 30, further comprisiag a bar-code reader means and a micro-well plate 
sensing means. 

39) The device for inspecting and classifying a plurality of microscopic crystals as in 
Claim 30, wherein said at least one computer means comprises a frame grabber 
means for receiving images of said plurality of microscopic crystals from said at 
least one camera means. 

40) The device for inspecting and classifying a plurality of microscopic crystals as in 
Claim 30, wherein said at least one coir^uter means is connected to a computer 
network means. 
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41) The device for inspecting and classifying a plurality of microscopic crystals as in 
Claim 40, wherein said at least one computer network means is the Ethernet, 
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301 



302 



Best Focus = 0 
BesL2 = -999 
Focus Error = 0 




308 




Initialization 



304 

<)^\/VindowiCenter 
Y Wlndow.Center 
xlWidth 
YJHelght 

NurrA)erjofJ2:_SlIces 
Z„Step_Slze 
.StarLZ value 



Move Z axis to Z_Posltion(N) = Slart_Z value 



314- 



N = N+1 



318^ 




Move Z axis to 
Z Position(N)-=Z„Position(N) + 
Z_Step„Size 


' i 





Acquire and 
Store lmage(N) 

1 



310 
■312 



Extract lmage(N) 
focus Value F(N) 
Subroutine 



316 




Best Focus = F(N) 
Best Z»N 



No 



326 




BesLZ* 
Numberj)f_Z_Sllces/2 

Focus §rror « 1 



Set Best Image = lmage(BesLZ) 
Set BestJmage.Z Z_Posltion(BesL2) 
Sefa Pointer Jo^BestJmage 



I 
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FIG. 31 
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Extract lmage(N) 
focus value F(N) 
Initialization 
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Pointer Jo_StoredJmage 
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Convolve Image(N) with Horz_Sobel Filter to find 
Horizontal Edges creating image(NLH 



Convolve lmage(N) with Vert„Sol)pi Fitter to find 
Vertical Edges creating lmage(NLV 



SUM lmage(NLH + lmage(NLV to create 
inriage(NLFoc"s 
Truncate values to range of 0 to 255 



Calculate Variance F(N) in 
!mage(NLFocus over the 
desired window 



return' ^ 
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Start 
Spedmen Auto 
[ Score & Classify j 
Subroutine 
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InNaiization 
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PoInter_to_BestJmage 
Type_of_ClassJficaaon 
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Calculate Average_Gray_Background normalized for welll type by 
multlpiyInQ by Mlcro^eli normalizer value 



X_BackGround_Center 
Y~BackGround_Center 
xlBackGround_Wldth 
Y_BackGround_Helght 
Micro-wen type Identifier 
Micro-well nonner value 



Calculate Averag6_Sray_ClassHy_WIndow 
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Calculate Diff_lmage_1 
If ABS[Plxel{x.y) - Pbce!(x+Diff.Sep.y+Oiff^Sep)] > BagLThresh 
THEN 

DlffJmageJ_Pbcel(x.y)= Sett 
ELSE 

Diff lmageJ_Pixel(x.y) = 0 
i 
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Calculate Dlffjma0e_2 
w If ABSpbcel(x+DifLSep.y) - Pb(el(x»y+DifLSep)] > 
\ Flag Thresti 

THEN 

Diff Image 2 Pixel(x,y)= SeH 
ELSE 

DIffJmage_2_Pixel(x,y) = 0 
: i 
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460 



Calculate Classifyjmage 
IfDIffJmage 1_Plxel{x,y) = Setl 
OR 

DlffJmage„2_PUe!{x,y)= Set1 
THEN 

ClasslV Iniage Pixei(x,y) = Set2 
ELSE 

aassffyJmage_Plxel(x,y) = 0 
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Calculate the nunnber_PixeIs_Set2 
Calculate Total„PixelsJn J/Vlndow 



Score„Gray = Av 


srage_Gray_.ClassIfy_Window/Aver 


age_Gray_Background 


SCOTB.RS 


ig = Number_Pixels_Set2/rotaLPi3( 


elsJnJ/Vlndow 
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CALL 

\Classlfy (Score_Gray, Score„Flag) subroutine 
FIG 34a step 468 
Type_of_Classification as parameter 



452, 



/ 



X_Ciasslfy_Center 

Y Classi^_Center 

xICiassHy Width 

Y_Classify_Hei^t 

Diff Sep 

Sett 

Set2 

Flag-Thresh 
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RETURN^ 
toRG36 
Step 852 
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FIG. 33 



17/26 



wo 03/034030 



PCT/US02/33150 



464 



468 




Calculate CLEAR 
If Score_Flag< Clear.Flag_LT 
AND 

Score Gray > CIear_Gray_GT 
THEN 
CLASS = 0 



484 



Calculate Light Precipitation 
If Score.FIag < Lgt_Precip_Flag_LT 
AND 

Sc6re_Flag > Lgt_Precip_Flag„GT 
AND 

Score Gray > Lgt Preclp_Gray_GT 
THEN 
CLASS = 1 



480 




Calculate Heavy Precipitation 
If Score^Flag < Heavy_Precip_FIag_LT 
AND 

Score_Flag > Heavy Predp_FlagjGT 
AND 

Score_Gray < Heavy Precip Gray^LT 
THEN 
CUVSS = 2 
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/ Clear_Flag_LT J 
i Clear_Gray_GT I 
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Lgt_Precip_Rag_LT 
Lgt_Precip_Rag_GT 
Lgt_Preclp_Gray_GT ' 
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Heavy_Precip_FlagLLT 
Heavy_Precip_Flag_GT 
Heavy_Precip_Gray_LT 
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Calculate Ugly Precipitation 
If Score Flag < LIgly„Preclp_Flag_LT 
AND 

Score Flag > Ugly_Precip_FlagLGT 
AND 

Score_Gray < Ugly_Preclp_Gray_LT 
THEN 
CLASS = 3 
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Calculate Micro_Crystals 
If Score^Flag > MIcrD_Cry_F!ag_GT 
AND 

Score Gray < Micro_Cry_Gray_LT 
THEN 
CLASS = 8 
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Mlcro_Cry_Rag_GT 
Mlcro_Cry_Gray_LT 
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Calculate Crystal 
If Score Rag < CrystaLFIag^LT 
AND 

Score Rag > CrystaLFIag^GT 
AND 

Score_Gray > CrystaLGray^GT 
THEN 
CLASS = 9 
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CrystaLFIag_LT 
CrystaLFIag_GT I 
CrystaLGray_GT \ 
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Calculate GRAiNY_Preclpation 
If CUSS = 8 
AND 

Score_Flag > Grainy_Flag_GT 
AND 

Score^Gray > Grainy_Gray_GT 
THEN 
CLASS = 7 




FIG. 34b 
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727 




\ Calculate a first set of Additional Image 

Features in Classffyjmage 
CALL Subroutine MVTJools_Blob_Analysis 
with pointer to Window in Classffyjmage 
Get 
numjound 
and for eadi found GET 
area(m), heiglit(m). wldth(m}, perimeter(m) 
location_X(m), locatlon_Y(m) 
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m = 1 
num_splierollte = 0 
spherolite(m) = 0 



Calculate Blob_Ratios 
Circle Uke_HW(m) » height(mywidth(m) 
Clrcle„Uke"AHW{m) = area(m)/(height(m)*width(m)) 



732 



No 



Calculate Spherollte 
If CLASSES 0R9 
AND 

Circle_Uke_HW(m) < ClrcIe„Uke_HW„Upper 
AND 

Clrcl9_Llke_HW(m) >Clrc!e.Uke_HW,Lower 
AND 

Circle.Uke AHW(m) < Circte_Llke_AHW_Upper 
AND 

Circle Uke_AHW(m) >Circle Uke J\HW_Lower 
THEN 

num_spherolite = num_spherolite +1 
m = m+1 
spherDlite{m) = 1 
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X_Classlfy_Center 
Y_ClassHy Center 
X_Classtfy Width 
Y_G!assify_Helght 
Pointsrjro_Ciassif/Jmag 
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FIG. 34c 
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Usl/ig location X{m). Iocation.Y(m), 1/2(heighum)}, 1/2(wi( 
calculate the average gray value within this reduce window for each 
spherollte previously found and set phase_sep_Gray(m) = to the 
found value 
then Nonmalize it by calculating 
phase_sep_Gray(m) =phase_sep_Gray(m) / 
.Average Gray Bactcground 
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phase_sep Gray > phase_sep„Gray_GT 
THEN 
CLASS = 4 
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phase_sep_Gray„GT | 



FIG. 34d 
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Analyze previously measured features from FIG 34c to 
further separate CLASS 9 Into sub-classes 9.0 through 
9.9 
set m = 0 

First test If CLASS = 9, ^f not then goto step 778 
Iftmethen 
CLASS = 9.0 
if num found =1 then goto step 768 
lfnotgoto769 



-YES 



•766 

(Average_Gray_BacKgr j 

ound / 
PointerJTo^BestJmage ' 

num found 
arealm), height(m), 
width(m), perimeter(m) 
location J((m) 
location_Y{m) 
num_spherolite 
spherolltefm) 
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771 



" m = 1 

HW.Ratio = Height(m) / Width 
\ (m) 
tf HW_Ratio > needle_HW_GT 

" or 
HW Ratio < needle_HW_LT 
^ttien CLASS = 9.2 

2 



if HW^Ratio > plate_HW_GT 
or 

HW_Ratio < plate_HW.LT 
then CLASS = 9.4 
THEN goto 773 ELSE goto step 778 



-YES- 



Uslng location_X(m).locatlon„Y{m). 1/2(helght{mB 1/2(>a^^^^^^^ 
calculate the average gray value within this "^^^^"^^^^^ 
previously found and set Chunlc„Gray{m) = to the found value 

then Nomnalize It by calculating 
Chunk_Gray{m) =Chunk_Gray(m) /Average.Gray.Background 
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\ If Chunic Gray < Chunl^_Gray_LT 
THEN 
CLASS = 9.6 



-|chunk_Gray_LT ^ 



k tfarea(m)>Chunk_50.GT 
THEN 
CLASS = 9.8 
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■(chunk,50,GT 



If area(m) > gorgeous.GT 
THEN 
CLASS = 9.9 
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^RBlum to FIG 33 
step 466 
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For m = 0 to numjound 
Calculate for each btob 




Average_Gray_BackgrouncI 

PoInterJTo.BestJmag© 

numjound 

area{m), helght(m), wldth{m), 

perimeter(m) 

location_X(m) 

location_Y(m) 

num^spherolite 

spherolite(m) ^ 
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m = m + 1 
HW Ratio =:Helght(m)/ Width 
(m) 

If HW_Ratio > needle_HW_GT 
" or 
HW_Ratio < needIe_HW_LT 
then CLASS = 9.1 
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needte_H\ALGT 
needle HW_LT 



If HW_Ratio > pIate_HW_GT 
or 

HW Ratio < plate.HW^LT 
ttien Ct-ASS = 9.3 
THEN goto 798 ELSE goto step 812 



— I — 
YES 
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plale HW_GT 
plateJHW.LT 
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Using location X(m). iocation„Y(m). 1/2(height(ni ) 1/2(widm(ni) ) 
calculate the average gray value within this reduce window for the Blob 
previously found and set Chunk_Gray(m) = to the found value 

then Nonnallze It by calculating 
Chunk Gray(m) =Chunk„Gray(m) /Average_Gray_Background 
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\ If Chunk_Gray.< Chunk_Gray_LT 
THEN 
CLASS = 9.5 



^804 
Chunk_Gray_LT / 



If area(m) > Chunk_50_GT 
THEN 
CLASS = 9.7 



^808 
- ^Cliunk_50,GT| 
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Return to FIG 33 
step 466 



23/26 



wo 03/034030 



PCT/US02/33150 



840 



( Main V_ 
I Program r" 
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/ inlHalteatlon Parameters 
I Inspection Lists 
\ Type^oLCIassification 
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Move to mlcro^ell within 
selected micro-well plate 








-NO— 1 
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YES> 
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acquire Image and locate 
droplet 
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Move to droplet wfthin 
micro-weli In micro-weil 
plate 



acquire droplet high- 
resolution image 
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CALL Specimen Auto Score & 
Classify Subroutine 
HG 33 step 440 
Type_of_Classlfication as 
parameter 




FIG. 36 
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